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" Austin's 5~year Title VII project was. intended to 
improve the achievement of . elementary students in_the following 
areas: oral language proficiency, knowledge of tasic concepts, 
treading ability in Spanish, and prcficiercy in English reading and 
math. Besults indicated that program participants gained zir knowledge 
of basic ccncepts at the kindergarten level and, to some extent, in 
Spanish reading ability. Fifth grade project students showed greater 
achievement than their noEprcject peers in English reading. However,, 
in fifth grade math and fourth grade reading and iiatb, project 
students ai>d nonproject students did net differ in tbeir rate of 
gains. The gap in achievement betueen Spanish-dominant or bilingual 
students and their Enalish-dominant peers remained. Ihe program 
raised several problems for consideration: (1) difficulty in locating 
appropriate instruments*! cr measuring achievement objectives in 
bilingual programs: (2t difficulty in' obtaining an appropriate sample 
of students: (3) personnel changes in both project staff arid 
evaluation staff during the ccurse of a project: and (4) lack of 
evaluaticn mcdels applicable for longitudinal evaluations oi 
achievement ir, bilingual programs. (CM) 
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EVALUATION OF ACHIEVEMENT OUTCOMES: AUSTIN'S EXPERIENCE 

The Austin Independent School District's five-year Titj.e VII project 
was concerned with" improving achievement of project participants in several 
major areas: oral language proficiency, knowledge of basic concepts, 
reading ability in Spanish, and proficiency in English reading and math. 

V 

This paper will focus on the specific achievement outcomes of Austin's 
project, and also point out those evaluation issues and problems which 
evolved for AISD during the project. 

Oral Langauge Development 

A primary objective of the AISD Title VII program was to provide oral 
language development in both English and Spanish. Measurement of this objec- 
tive was conducted using the Oral Language Proficiency Measure (PAL Test) 
developed by the El Paso Public Schools. Each year, the test was administered 
in the fall by school personnel to kindergartners with a language other than 
English spoken in their homes. Spring posttesting for a sample of project 
and nonproject kindergartners was conducted by trained evaluation staff. 

Comparisons of gains made by project and nonproject students were 
made using a regression approach in which the criterion variable was the 
spring posttest score on English or Spanish and the predictor variables were 
the students' project status and pretest scores. In nontechnical terms, 
project and nonproject students were compared on posttest scores, while 
adjusting for initial differences between the two groups. Reautta oh tkeAZ 
compasuAonb Ojj pnojzcX and nonpAojtcX 6tudtnt6 Indiattzd thcut ntitkitt QKoap 



Jihomd con&litmtiy qiexxtoi gcutu 'in EnyU&k on. SpcutUk PAL 4Co*ea, acAou 
WtcA ojj tkz j^tye ytoM oi tkz pLOjttt. (Bigure 1 illustrates these longi- 

^^^^ 4 

Gliding Hidings. ) > ^ 

Several problems .exist with' interpreting these findings, of course. 
The fcAL is subjectively scored, as were all oral language measures availabi 
for use in primary grades at the conception of this project. In addition, 
the students were not randomly assigned to project and nonproject conrrol 
groups, a problem which is present throughout all the achievement findings 
presented here. 
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* Resulted from a drop In Spanish PAL scores for nonproject students, 
since project students actually- showed no gains. 

Figure 1. COMPARISON OF PROJECT AND NONPROJECT KINDERGARTENERS ON GAINS 
MADE IN ENGLISH AND SPANISH ON THE PAL, ACROSS FIVE YEARS OF 
THE TITLE VII PROJECT. 



Knowledge of Basic Concepts , 

A second objective of the Title VII project in the AISD was to 
improve students* knowledge of the basic concepts which are considered 
important at the early grade levels. Project and nonproject kindergarten 



students were administered the Boehm Test of Basic Concepts in *the fail _ * 
and spring each year. Project students who were Spanish-dominant were 
tested with the Spanish version of the Boehm, while all others were tested 
in English. Comparison of project and nonproject students tested in Eriglish 
Indicated that for 'the last three years of the five-year project , ptoj&ct 
Attidtnte gained mate than nonptojezt ±£ud&yvt6&p 

Unfortunately, no honprpje^t ^students were tested in Spanish and some* 
problems exist in comparing the gains of students tested in Spanish with 
those of other students. It appears that gains of project students tested 
in Spanish were comparable to gains of English-tested project students. 
However, it is unclear whether, scores on the Spanish version of the test 
are equivalent to scores on the English version; in other words*, are the 
tests of equal difficulty? The Spanish version of the Boehm is a direct 
translation of the English version, but that process of translation does 
not ertsure that the questions are worded in~such a wajr as to be of equivalent 
difficulty. However, there is obviously a need for instruments to measure 
achievement in content areas that have equivalent forms in English and Spanish, 
especially at the lower grade levels. 
Improvement itu|jpanish Reading 

Another objective of the AISD Title VII project was to develop reading 
skills in Spanish for project students. A sample of project and nonproject 
students in grades 2-5 was tested with' the Spanish Reading Test or Prueba de 
lectura in the spring of each year. Achievement gains in Spanish reading 
were then compared for project and nonproject students. The results of 
these comparisons indicated that ptoj&ct AtudnntA Ahoufcd 4ma£X, bat dOnhJUtWt 

) 

t 



qouu in tkZAji Spcutuh zieoiiotg, and #ieie gain* urc/ie 3*-&i£&i than thott ofi 
nonpfiojzct 4tudewU. 

In Interpreting these results, some caution seems needed. Out of a* 
possible 134~ points, mean gains for fourth graders after three years of" 
instruction equalled only 12.40 raj* score points, while mean gains for 
fifth graders with three or four years of instruction were 12.10 raw score 
points. The fifth-grade cohort began second-grade with higher scores in 
Spanish reading than did the fourth grade group. The gain in raw scores for 
-these groups resulted in mean Spanish reading scores for fourth graders that 
were at only the 12th percentile, while fifth graders were scoring at only 
the 26th percentile, using national norms. (Texas, norms were not available 
for the form and level of test which was .used.) 
Achievement in English Reading and Math 

A major objective of Austin's Title VII Bilingual Project was to 
Improve the achievement of project students in English reading and math. 
In an effort to minimize the extra testing involved for the students,' 
this objective was measured using*the same test adopted by the District * 
for its annual spring testing effdw^—the California Achievement Tests 
(CAT, 1970). % 

Although project students across the years showed consistent gains 
in their Reading and Hath Achievement Development Scale Scores on the CAT, 
perhaps the most interesting comparisons are between longitudinal gains of 
project and nonproject students. A sample of fifth graders was chosen who 
had consistently participated or not participated in the Title VII project 



since first grade. Compa/uAonA o£ pkojtct and nonpxjojtct ^i^th-gfvsdz 
ttudtnts indicated that the. pKojtct 6tudznt6 thawed bi%nihicaft£ltf g/i&tizt 
$auu than nonp^oject 6tudmt& on the CAT in leading, but no significant 
-iiMwftttceA_ wc/te found in, math. 

A timittvi sample of founth-gtuxdz studmU who u>e*e con* intently 
pAo/ect o>t nonp>t0/e.c^ KZvtodLzd no significant di&fvivwiQ. -oi Kuxding ok 
math goots on the. CAT. ^ ^ 

Another interesting comparison of achievement gains was of groups of v 
project students who differed with respect to language dominance. Upon 
entry into school, project students were classified on the basis of their 
PAL scores as English monolingual, English dominant, or Spanish" dominant/ 
bilingual. An examination of the achievement test scores for these three 
language groups at the first grade-indicated that means for Spanish- 
dominant/ bilingual students were consistently lower than those of the 
other two groups. After four or five years of participation in the project, 
the gap in achievement between the Spanish-dominant /bilingual' students and 
their English-dominant or English monolingual peers is not closing. In 
fact, at fifth grade the gap was significantly wider than it had been atr 
first grade, with Spanish-dominant/biliagual students falling further behind 
in both reading and math. (Figures 2-5 graphically depict these results.) 

Several factors need to be considered Jit interpreting these results. 
There was a high attrition rate among the original samples of students, 
and it is unclear what impact short-term participation in the project might 
have had on these students. In addition, the District .had a policy until 
1979 of exempting special education students from testing with the CAT,* 
and thus the sample of students with valid scores in earlier years of the 




FIRST GRA OE FIFTH GRADE 

AOSS SCOR2 AOSS SCORE 



Figure 2 . READING ACHIEVEMENT GAINS OF PROJECT STUDENTS ACCOSSIKG TO 



LANGUAGE DOMHtAiKE (OtAOE 5)* 
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Figure 3, "MATH ACHIEVEMENT GAINS 07 PROJECT 3TTOENTS ACCORDING TO 
LANC0AGZ MMTirANC»_J^AIlS 5^, ^ 
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project did not reflect the true population of students In the project 

and nonproject classrooms* This policy may have resulted In distorted 

i . . - 

.estimates of group means. s v 

Arguments by bilingual educators also state that if the CAT is culturally 
biased in favor on the Anglo student, benefits to project students may have 
been masked. Finally , learning in content areas may have occurred with the, - 
Spanish-dominant/ bilingual students that is not reflected in these results* ' 
Summary of Achievement Data * . 

These results indicate that participants iivthe program received 
benefits in terms of knowledge of basic concepts at the kindergarten level 
^and also to some extent in Spanish reading lability* In addition, fifth-grade 
project students out gained their nonproject peers in English reading, although 
these gains appear to be due to the gains of English-dominant and English- 
monolingual students. In fifth-grade math and fourth-grade reading and math, 
project students'and nonproject students did not differ in their rate of 
gains* The gap in achievement between Spanish-dominant ^or bilingual students 

and their English-dominant peers does not appear to be closing. 

* * •* 
Summary of Evaluation Issues and Considerations 

. ^ + 

j Several major issues arise for further consideration, by evaluators of 

bilingual programs* First,' there are serious difficulties in locating 
appropriate instruments for measure g achievement objectives in bilingual 
program**. Language instruments Are frequently subjective and many instru- 
ments measuring content are lacking in tnf equivalent English/Spaaish 
forms. In addition, potential cultural bias in English achievement tests 
continues to be a problem. 

• Second, it is often difficult to obtain an appropriate sample of 
students, <lu order to accurately assess the objectives. 

10 



Due to nonrandom assignment of student* in this and many otfier programs* 
/Control 11 groups may be nonexistent or at best* not comparable to the 
students participating in the*project. A related sample issue is that 
longitudinal" gains become very difficult to measure because of the high * 
attrition rate among participants—families migrate, districts move 
students in desegregation efforts, and schools nay elect to drop out of 
the project^ as happened in AISEL In short, in order to control for this 
attrition« initial* samples must -be large if longitudinal studies are to 
be successful. - , 

Another problem concerns personnel changes In both project staff and 
evaluation staff during the course of a long-term project. Turnover among 
-teachers or administrative j^tgff of the project means that it is difficult 
to assure that students receive a consistent program. Students within the 
same project may have' different opportunities to learn, and it is not easy to 
measure -such differences. Changes in evaluation staff can result in loss 
of information about trends* as well as a potential lack "of consistency in 
the way objectives' are measured. 

Finally! there is a lack of evaluation models applicable for longitudinal 
evaluations of achievement in bilingual programs. In addition, there are still 
many questions concerning the best ways *ta .measure change over long periods 
of time. Development of su^fi models of evaluation and measurement would have 

V 

the advantage of allowing comparisons between and across programs, as veil as 
more satisfactory evaluation of individual programs. 



